Auditory-visual Interactions

Most displays or other devices designed to communicate information focus on a single sensory modality. This is perhaps a consequence of the fact that most research examining human perceptual capabilities has focused on a single sensory system at a time. However, most events in the natural environment generate physical information affecting multiple sensory modalities. This information is typically co-located in both space and time, and our perceptual systems have evolved to create a single coherent representation of our environment. Recent research has increasingly acknowledged the importance of these regularities, and the topic of multisensory integration became critical for understanding global awareness of the environment. This chapter presents an overview of the most recent research into the integration of auditory and visual information and presents considerations for the design of multisensory (i.e., auditory-visual) displays. The initial section of the chapter presents comparison of the basic features of the auditory and visual modalities and examines the effects of interacting visual and auditory stimuli. Included in this section is a discussion of the relative strengths and weaknesses of each modality and the situations in which one modality dominates. This section is followed by a discussion of perceptual effects when information from both modalities is complementary or conflicting and the role of multisensory attention on the perception of auditory-visual information. The final section of the chapter presents a discussion of the applications of auditory-visual integration principles to audio-visual display (e.g., helmet-mounted displays [HMDs]) and signal designs. Research on the interaction of a third modality, tactile or haptic displays, is just beginning and will not be discussed extensively here, although many of the same considerations can be expected to apply (see Chapter 18, Exploring the Tactile Modality for HMDs).